Method and system to analyze sports motions using motion sensors of a mobile device

ABSTRACT

A method and system to analyze sports motions using the motion sensors of a mobile device, such as a smart phone, is provided. This method uses the mobile device motion sensor output to define the impact point with a virtual object, such as a golf ball, baseball or tennis ball. The motion sensor signature of the sports motion is analyzed for characteristics, specific to each type of sports motion. A method is disclosed using multiple sensors outputs in a mobile device to compute the impact point with a virtual object, such as a golf ball, baseball, tennis ball. Further, a method is disclosed where moving virtual sports objects interact with virtual sports motions and the responsive outputs are displayed on the mobile device and/or any Web-enabled display device.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority of provisional applications Ser.No. 61/551,388 to Jeffery et al., entitled “Using a Mobile Phone WithIntegrated Motion Sensing For Evaluation of Sports Motions and ProvidingCustomized Sports Instructions Responsive to Said Evaluation”, filed onOct. 25, 2011; Ser. No. 61/580,534 to Jeffery et al., entitled “Using aMobile Phone With Integrated Motion Sensing For Golf Swing Evaluationand Customized Golf Club Fitting”, filed on Dec. 27, 2011; and Ser. No.61/713,813, to Jeffery et al., entitled “Method to Analyze SportsMotions Using Multiple Sensor Information From a Mobile Device”, filedon Oct. 15, 2012; each of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a method and system to analyze sportsmotions using motion sensors of a mobile device.

2. Description of the Related Art

There is extensive prior art in the evaluation of sports motions,particularly golf swings. For example, several manufactures providelaunch monitors which use high speed video cameras to analyze the launchangle and velocity of a golf ball. Additionally, there is disclosure inthe prior art of the use of various devices attachable to a golf club tocapture motion data for further analysis. For instance, WIPO PublicationNo. WO2011085494 to Hashimoto et al. describes a system of motionsensors attachable to a physical golf club with data analyzed by apersonal computer. Similarly, U.S. Published Patent Application No.2005/0054457 to Eyestone et al. discloses a clip-on motion sensor thatcan attach to a physical golf club with data collection on a device witha Bluetooth connection. Furthermore, PING, Inc. has developed a puttinganalysis system wherein users hit physical balls while putting, andmotion sensors in a mobile phone clipped onto the physical golf clubprovide data used by an application to analyze the putt, and to compareresults to that of professional golfers. However, such prior art systemsand methods require impact with a physical ball, and/or use of, orattachment to, sports equipment.

In the world of computer gaming, several sports-related games have beenintroduced that utilize motion sensors. For example, the Nintendo Wii isa popular gaming console, and the Wii controller contains a 3-axisaccelerometer connected via infrared to a gaming console. The NintendoWii controller is connected via a Bluetooth connection to the gamingconsole and senses acceleration in three axes using the accelerometer.The Wii controller also features a PixArt optical sensor, which incombination with a 10 LED sensor bar placed several feet from the userand physically connected to the game console, allows the determinationof where the Wii remote is pointing. An add-on to the Wii controller isavailable which includes a gyroscope, but with or without, the Wiirequires the user to purchase an entire gaming system which isexpensive, cumbersome, and requires physical attachment to a television.

SUMMARY OF THE INVENTION

One aspect of the disclosure relates to a method of analyzing sportsmotions. The method comprises determining a starting point of a sportsmotion to be simulated using a mobile device having integrated motionsensors; moving the mobile device from the starting point along a pathto complete the simulation; obtaining motion data from the motionsensors relating to the starting point and the movement along the path;determining an occurrence of a simulated sports event using the obtainedmotion data; and outputting information related to the simulated sportsmotion; wherein the mobile device is not attached to any piece of sportsequipment and the starting point is not indicated by user-entered input.In an embodiment, the starting point is indicated by the mobile devicebeing held still for a predetermined length of time. In an embodiment,the information related to the simulated sports event includesoutputting the information on a screen of the mobile device, forexample. In another embodiment, the information related to the simulatedsports event includes outputting the information to a gaming system.

In an embodiment, the sports event that is determined is the impactpoint with a virtual object, such as a virtual golf ball. In the case ofgolf, the mobile device would be used in place of a golf club tosimulate a full golf swing, a chip, or a putt. In an embodimentspecifically for a full golf swing, the impact point is determined byfinding a moment when the pitch of the mobile device is at or near asecond minimum. In another embodiment, the impact point is determined byfinding a moment when the pitch of the mobile device is at or near aminimum and the derivative (rate of change) of yaw of the mobile deviceis at or near a maximum.

The method of analyzing sports motions further can include the step ofdetermining the velocity of the virtual object around the impact pointusing gyroscope data (without requiring an accelerometer) along withdata for the estimated arm length, club length, and arc length for theswing type. In another embodiment, the instantaneous velocity of avirtual object can be determined using a suitable accelerometer. In anembodiment, a multiplier can be applied to the velocity based onestimated wrist hinge and forearm rotation as measured by yaw and roll,respectively, of the mobile device at the impact point, included in theobtained motion data.

The method of analyzing sports motions can include the step of analyzingthe simulated sports motion. Analyzing the simulated sports motion caninclude analyzing the pitch of the mobile device during the simulatedsports motion and the roll of the mobile device at an impact point, forgolf for example. Analytical information regarding the sports motion(e.g., golf swing) can be presented on the mobile device for the user,or a Web-enabled display connected to a cloud-based server via theInternet.

In an embodiment, the sports event is a release point, e.g., the releasepoint of a bowling ball, a lacrosse handle, a basketball, a baseball, ahockey stick, a bean bag, and a fishing rod.

The method can be further extended to include sport motions involvinghitting virtual object(s) that are moving (such as a baseball, a hockeypuck, a tennis ball, a ping pong ball, a shuttle cock, etc.) withvirtual sports equipment (such as a baseball bat, a hockey stick, atennis racquet, a ping pong paddle, a badminton racket, etc.) The methodcan also be used to analyze sport motions that result in impact withobjects at some distance from the player, such as a virtual fly fishinghook cast towards a fish, a virtual bowling ball striking bowling pins,a virtual basketball thrown at a virtual hoop, a virtual Americanfootball thrown at a running virtual receiver, a virtual bean bag or avirtual javelin thrown at a target.

Another aspect of the disclosure relates to a method of analyzing sportsmotions. The method comprises (a) displaying a moving virtual object ona screen of a web-enabled display;

(b) obtaining motion data from motion sensors of a mobile device, themotion data relating to a simulated sports motion; (c) determining astarting point of the sports motion and movements along a path of thesports motion, using the obtained motion data; (d) comparing timing ofthe simulated sports motion with that of the moving virtual object todetermine whether the virtual object was impacted, and if so, the impactpoint; and (e) outputting information related to the simulated sportsmotion; (f) wherein the web-enabled display is a separate and distinctdevice from the mobile device and viewable by a user of the mobiledevice as the user moves the mobile device to simulate the sportsmotion; the web-enabled display and the mobile device are connected viathe Internet to a cloud-based engine; and the cloud-based engine managesgame playing and is at least partly responsible for performance of step(d). The web-enabled display can be one of a computer, a tablet, aweb-enabled television, and another mobile device that is connectable tothe Internet. In an embodiment, the game involves a thrown or hitobject, and the virtual object is one of a baseball, a tennis ball, aracquet ball, a ping pong ball, a hockey puck, and a badminton shuttlecock. Displaying the virtual object can include displaying a video, asprite, a cinema-graph, or an animation on the web-enabled display. Inan embodiment, clocks of the cloud-based engine, the web-enableddisplay, and the mobile device are synchronized using the Network TimeProtocol (NTP). In an embodiment, comparing timing includes usinginterpolation to determine the impact point.

In an embodiment, the method further comprises (g) displaying a virtualobject flight on the web-enabled display responsive to the simulatedsports motion. The virtual object flight can be a golf ball flight, abaseball flight, a tennis ball flight, or a ping pong flight, forexample. In an embodiment, the method further comprises (h) displayinginteraction with the virtual object flight on the web-enabled display.Such interaction can include catching or hitting the virtual object,which can be displayed using one or more of a video, a sprite, acinema-graph, and an animation. The interaction can include the virtualobject being hit back to the user, such that after step (h), steps (b)through (f) can be repeated to provide a game playing experience.

Another aspect of the disclosure relates to a system for analyzingsports motions, comprising a mobile device including a memory, aprocessor, and an integrated multi-axis gyroscope. The mobile device isconfigured to determine a starting point of a simulated sports motion;obtain gyroscope measurements from the gyroscope relating to orientationof the mobile device at the starting point and during movement along apath simulating the sports motion; determine an impact point with avirtual object and velocity of the mobile device around the impact pointusing the obtained gyroscope measurements; and output informationrelated to the simulated sports motion. In an embodiment, the mobiledevice is attached to an ancillary device (e.g., a golf club or weightedgolf grip).

Another aspect of the disclosure relates to a system for analyzingsports motions, comprising a mobile device including a memory, aprocessor, and integrated motion sensors. The mobile device isconfigured to determine a starting point of a simulated sports motion,the starting point indicated by the mobile device being held still for apredetermined length of time; obtain motion data from the motion sensorsrelating to orientation of the mobile device at the starting point andduring movement along a path simulating the sports motion; determine animpact point with a virtual object using the obtained motion datarelated to the orientation of the mobile device along at least two axes;and output information related to the simulated sports motion via themobile device. In an embodiment, the mobile device is attached to anancillary device.

Other aspects and embodiments of the invention are also contemplated.The foregoing summary and the following detailed description are notmeant to restrict the invention to any particular embodiment but aremerely meant to describe some embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates various types of rotational movement measured byinternal motion sensors of a mobile device;

FIG. 2 illustrates a system to analyze sports motions using the motionsensors of the mobile device;

FIG. 3 illustrates the pitch and roll of the mobile device during anexample full golf swing useful for determining swing accuracy;

FIG. 4( a) illustrates a technique for calculating club head speed for afull golf swing;

FIG. 4( b) illustrates a technique for calculating club head speed for agolf chip;

FIG. 5 illustrates use of pitch data of the mobile device to determinethe impact point and the speed of the club head through the impactpoint;

FIG. 6 illustrates use of pitch and yaw of the mobile device during anexample full golf swing to determine the impact point;

FIGS. 7( a) to (c) illustrate changes in pitch of the mobile deviceduring an example full golf swing, chip, and putt, respectively;

FIG. 8 illustrates changes in pitch, yaw and roll of the mobile deviceduring an example tennis swing or seated horizontal golf swing;

FIG. 9 illustrates yaw, roll and pitch for a baseball swing;

FIG. 10 illustrates pitch, roll and yaw for a bowling motion;

FIG. 11 illustrates an example mobile device holder mounted to anancillary device; and

FIGS. 12( a) to (c) illustrates the mobile device holder of FIG. 11being placed onto the ancillary device.

DETAILED DESCRIPTION OF THE INVENTION

As used herein, a mobile device refers to a hand-held device having amicroprocessor, memory, and integrated motion sensors.

As used herein, calibration point refers to the location in time andspace of the mobile device in a set-up position prior to the start ofthe sports motion.

As used herein, impact point refers to the location in time and space ofimpact with a virtual object.

As used herein, release point refers to the location in time and spaceof the release of a virtual object.

In some sports, such as golf, the impact point and release point may bethe same physical location in time and space. However, in other sportsthe impact point and release point may not coincide. For example, inlacrosse the ball is caught by a long-handled stick (the ball impactsthe lacrosse stick at the impact point) and then is later thrown fromthe stick from a different location (at the release point). Furthermore,in some sports there is only a release point since there is not a pointof impact. For example, in fly fishing the release point occurs by aflick of the wrist imparting angular momentum to the fishing rod, whichabove a certain maximum value causes the weighted fish hook to releasefrom the rod.

FIG. 1 illustrates the various types of rotational movement measured byinternal motion sensors of a mobile device 10 such as an Apple iPhone4Gs, an Apple iPod Touch, or a Samsung Galaxy III smart phone. Thesesensors of the mobile device 10 include an accelerometer to capture X, Yand Z acceleration data (expressed in G's along a respective axis), anda gyroscope to measure pitch, roll and yaw of the mobile device 10 as itmoves (expressed in radians with respect to a respective axis). Atpresent, the motion sensors sample at about 100 times per second (100hertz), with this data made available (by either polling or having thedata pushed) to an application program loaded on the mobile device 10. Arepresentative gyroscope useable in conjunction with the presentinvention is the L3G4200D gyroscope made by STMicroelectronics, Inc.However, it is to be understood that the present invention is notlimited to motion sensor technology currently available.

FIG. 2 illustrates a system for analyzing sports motions 100. As shown,the system for analyzing sports motions 100 includes an analysis engine50 which can include an application program loaded in memory of themobile device 10. Such an application program for an Apple iPhone can bedeveloped using the Apple Developer Suite, including use of Xcode,Interface Builder, and iPhone Simulator development tools, or via customprogramming in Objective C. The Core Motion framework can be used toobtain and manage the accelerometer and gyroscope data. It is to beunderstood that where the mobile device 10 is other than an AppleiPhone, other programming techniques and tools can be used. For example,where the mobile device 10 is a smart phone utilizing the Androidoperating system, an appropriate Android software development kit (SDK)can be used to provide the tools and application program interfaces(API) for developing the application on the Android platform using theJava programming language.

A notable feature of the system for analyzing sports motions 100 is thatinitially (1) the user holds the mobile device 10 still for apredetermined length of time (e.g., at least 1 second for golf), whichrepresents the “zero” calibration of the mobile device 10, this is thecalibration point for the sports motion. The set-up position of theplayer and calibration point will be different for different sports: forgolf the set-up is the address position of the golf swing, for baseballit is the location of the ball on a virtual tee, in tennis or tabletennis, it can be the at rest position for the racquet before swinging.

Note that the calibration point need not be a specific setup positionwhere the mobile device 10 is held still for one second in the sameposition. For tennis and table tennis the calibration point may beobtained from any point where the player has his or her hands in a readyposition to play, and/or may be selected as an end point or calibrationpoint of a prior sports motion.

Next, the user takes a swing (2) and the motion sensor data is obtainedby the analysis engine 50 during the swing (by polling, for example).The analysis engine 50 next detects the swing type (3). The swing typecan either be entered by the user via a screen of the mobile device 10(for example, selection from a pull-down menu listing ‘full golf swing’,‘chip’ ‘putt’) or detected based on the motion signature, and, perhapsalso another factor such as, in the case of golf, club selection (e.g.,a wedge is selected and the motion signature shows that the user onlyswung back 30 degrees so the type of swing is a chip). Next, theanalysis engine 50 detects impact with a virtual object and/or therelease point (4), analyzes the motion signature to calculate the motionvariables (for golf, the swing speed and accuracy of the swing) (5), anddetermines the key output parameters (for golf, the ball flightdistance) (6). Finally, information regarding the swing can be displayedon the graphical output on the mobile device 10 (7) or a separateweb-enabled display. Alternatively, or in addition, informationregarding the swing can be used as inputs to a gaming system, such as isdisclosed in co-pending Ser. No. 61/641,825 to Jeffery et al., entitled“Web-Based Sports Game Platform With Mobile Phone Motion Sensor Input”,filed on May 2, 2012, which is incorporated herein by reference.

These and other novel elements of the invention will become apparentfrom the following detailed description of the invention in the contextof a golf swing sports motion and then with respect to other sportsmotions including baseball and bowling. However, it is to be understoodthat the following examples are not meant to be limiting.

Golf Example

FIG. 3 illustrates the pitch and roll of the mobile device 10 during anexample full golf swing. An important element of the present inventionis the calibration of the mobile device 10 by holding the mobile device10 still in the address position (position 1). The motion signature forthe pitch then increases in the backswing (position 2) and has a localminimum at the top of the golf backswing (position 3). However, theminimum (position 3) is an artifact of the pitch motion sensor rotatingmore than 180 degrees. In actuality, the pitch continues to increase toa maximum, greater than 180 degrees, at the top of the backswing.However, limitations of the sensor constrain the motion signature to 0to 180 degrees. The pitch data continues to decrease in the downswing(position 4), back to the impact point (position 5), as shown.

Accuracy Analysis

Note that at the impact point (position 5), the mobile device 10 hasreturned to near the initial calibration point (position 1), which forgolf is the hand position at impact with a virtual golf ball and a localminimum. For a high speed golf swing the minimum at the impact pointdoes not return exactly to the calibration zero due to resolution limitsof the gyroscope. Determining the impact point is of vital importancebecause the roll of the mobile device 10 at this point defines the hookor slice of the club. In other sports, the impact point is vital indetermining the hook and slice of a bat or a racquet, and/or the releasepoint in throwing or casting sports. From the impact point, the golfswing continues through follow through, positions (6) and (7).

In summary, pitch data, or the rotation around the axis that cuts themobile device 10 into top and bottom halves when looking at the screen(X-axis) (see FIG. 1) is the most telling data stream as a golfer movesthrough their swing. Impact can be found at the major minimum thatapproaches the starting calibration point (which is defined as “zero” bytaking the average of all phone position/orientation data over thecourse of one second, for example, taken prior to the swing when thegolfer is in their set-up position). To bring context, in a golfer'sswing, pitch data rises as the golfer goes into their backswing, returnsto calibration as he or she swing through impact, then rises again as heor she moves into their follow through. Impact is the pitch positionthat gets closest to the set-up, or calibration point.

In an embodiment, the impact point for a full golf swing is selected tobe the second minimum of pitch using a crawler algorithm. In anotherembodiment, the minimum can be confirmed by aligning it with a spike inZ-acceleration. When more than one major minimum in pitch is found, theminimum selected as impact is determined by which point has the greatestZ-acceleration. This confirmation helps in cases where a golfer'sbackswing or follow-through rotation is so great (near 360 degreerotation from set-up) that the gyroscope flips completely and createsextra minimums near calibration.

Once impact is found, swing accuracy is determined by subtracting rolldata at impact from roll data at calibration. Roll data, or the rotationaround the axis that cuts the phone into left and right halves whenlooking at the screen (Y-axis) describes “open and closed” facepositions on the club head. FIG. 3 shows an expanded view of the rolldata. Swings that return a negative difference mean that the userover-rotated at impact which implies a closed face at impact and aresulting draw or hook depending on the amount. Swings that return apositive difference mean that the user under-rotated at impact whichimplies an open face at impact and a resulting fade or slice. Swingsthat return a near zero value mean the club face very closely matchedcalibration orientation at impact and imply a straight ball flight.

Speed Analysis

Club head speed is a critical parameter for golf in defining the ballflight distance. The golf club manufacturers have empirical tables whichdetail the ball flight distance for golf balls hit by club heads movingat a specific swing speeds. Such tables also take into consideration theclub type (e.g., driver, 5-iron, putter), the club head loft, the shaftstiffness, and other variables that impact the ball flight.

Swing speed is a complex calculation due to the mechanics of sportsmotions. The challenge is that the sensors measure motions of the handswhereas we are interested in calculating the speed of virtual sportsequipment, such as a golf club head. Extensive experiments withprofessional athletes were conducted using appropriately fitted sportsequipment to understand how hand and arm motions translate to the motionsensor data outputs. While the analysis for golf is illustrated, it isto be appreciated that the present method is generalizable to othersports motions, such as those found in the sports of baseball, tennis,bowling, basketball, American football and table tennis.

FIG. 4 illustrates the swing motion elements for (a) a golf full swingand (b) a chip, which is a short swing. If the club is swung exactly inline with the arms, then the mobile device velocity, V, is related tothe club head velocity (V_(club head)) by:V _(club head) =V×(Arm Length+Club Length)/Arm Length  (1)

However, expert players hinge their wrist and rotate their forearms toincrease the velocity of the club head through the ball. These hingingand rotating motions can dramatically increase the velocity of the clubhead through impact, so that Equation (1) is a gross under estimate ofthe golf swing speed for most golfers. It is a good for putting,however, since there is no hinging of the wrists.

FIG. 5 illustrates specifically how we calculate the speed of the mobiledevice 10 for a golf swing. The motion signature for the pitch of themobile device 10 for an example full golf swing is graphicallyillustrated. Shown below is the corresponding sports motion with points(4), (5) and (6) in pitch data labeled on the swing. We first find theimpact point in pitch data, defined as the local minimum of pitch at thebottom of the swing (point 5). We then look forward and back in pitchdata by 60 degrees. These data points, assuming proper wrist hinging,align with positions in the swing (4) and (6). Generally, about onetenth of a second passes between these two positions, so that given theplayer's arm length we can find the mobile device speed 10 around impactby dividing the length of a 120 degree arc where the radius of the arcis equal to the arm length by the amount of time passed: This deliversthe speed of the mobile device 10 (hand speed). A similar method can beused for chipping but with a shorter arc length of 55 degrees or lessdue to the reduced swing length.

It has been found, using high speed video clocking, that the driver clubhead speed can be as slow as 2.4 times hand speed (this is in the caseof a user swinging a club with rigid arms, forearms, and wrists) or asfast as 6 times hand speed (in the case of a world class professionalgolfer). The difference between these two multipliers comes from thecombination of forearm rotation and wrist hinge which allow golfers toforce the club head to travel through a much greater arc length(sometimes even close to 180 degrees) in the time it takes the hands totravel through the 90 degrees of arc length around impact. Themultiplier we choose is driven directly by gyroscope accelerationthrough impact on the Z and Y axis (yaw and roll) which account forwrist hinge and forearm rotation respectively.

From detailed experiments with the iPhone 4 and 4s it was found that thegyroscope is particularly accurate, so that the roll data is very goodto predict hook or slice within approximately half a degree. Theaccelerometer data from the iPhone 4, however, is “noisy”, and is notparticularly accurate over the entire golf swing, but does work well formeasuring forearm rotation rate around impact. This is why we divide theswing into portions and calculate an average velocity, V, of the mobiledevice through impact:

$\begin{matrix}{V = \frac{D_{2} - D_{1}}{t_{2} - t_{1}}} & (2)\end{matrix}$where D₂−D₁ is the distance between points (4) and (6) in FIG. 5; andt₂−t₁ is the time taken to cover the distance D₂−D₁. A shorter distanceis preferred, since this enables a closer approximation of theinstantaneous velocity at the impact point. However the 0.01 secresolution of the current gyroscope requires us to use the 120 degreearc. In the future, as the sampling resolution of the gyroscopeincreases, a 30 degree arc or less will be preferred.

Equation (2) is an approximation of the actual instantaneous velocity ofthe phone, and is only a first order approximation of the speed of thegolf club head, since it does not include the wrist hinge or forearmrotation described above. Via detailed experiments with a high-speedvideo camera we were able to find multipliers for these variables, withthe result of calculating club head speed within +/−10% for a variety ofswing types. From club head speed we can predict ball flight distance inideal conditions.

We envision that the data quality output from the accelerometer willimprove dramatically in future versions of iPhone or Android basedphones. In an embodiment of the present invention, the velocity of amobile device 10 (having a sufficiently accurate accelerometer) atimpact is calculated by integrating the acceleration from the top of thebackswing (t_(bs)) to the zero (t₀) of the mobile device:V _(x)=∫_(t) _(bs) ^(t) ⁰ a _(x) dxV _(y)=∫_(t) _(bs) ^(t) ⁰ a _(y) dyV _(z)=∫_(t) _(bs) ^(t) ⁰ a _(z) dzwith the total mobile device velocity at impact:V=√{square root over (V _(x) ² +V _(y) ² +V _(z) ²)}  (4)where t₀−t_(bs) is the time between the minimal at the top of the backswing (t_(bs)) measured from the pitch data and the zero at the bottomof the swing at impact, t₀. The integrals are calculating in thesoftware using a fourth order Runge-Kutta algorithm. See for example,William H. Press et al, Numerical Recipes 3rd Edition: The Art ofScientific Computing, 2007.

The velocity component vectors (4) are difficult to accurately calculatewith the current version of the accelerometers, since the internalaccelerometer has a noisy output, hence why we currently use the averagemethod equation (2). Data on the swing motion is presented to the userand stored, local to the app and on a server in the user's account, forlongitudinal comparisons of swing consistency improvement.

The user can also attach the phone to their golf club via a cradle andcompare actual practice swings to the computed swings for distance andaccuracy. We use a similar analysis when the phone is attached to theclub, but the multipliers are different primarily due to users swingingthe golf club slower than the phone, the phone is lighter than a golfclub so one's hands naturally go faster.

As an additional example of swing analysis we consider putting, ratherthan the full swing of a golf club. PING, Inc. has previously created aniPhone App for putting. Their prior art invention has three significantlimitations however: Their method (1) requires an attachment to aputter, (2) requires impact with a physical ball, and (3) is notaccurate for long putts (greater than approximately 20 feet).

The method described herein does not have any of these limitations.Similar to the full swing described above, the user holds the mobiledevice 10 as if it were a putter, and after one second of being heldstill it vibrates: the phone is ready. The user then putts an imaginary(virtual) ball. Compared to the full swing, the pitch data from thephone is now a relatively smooth sine wave function with a minimum atimpact. The putter stroke is analyzed similar to the full golf swing,but with average velocity calculated from Eq. 2 where D₁ and D₂ are therespective maximum distances pull back and stroke through impact withthe ball. An advantage of the putter stroke is that the function issmooth and the speed is relatively slow compared to the full golf swing.Hence, equations (3) and (4) can also be used to calculate aninstantaneous velocity at impact—we use both methods, integration ofequations (3) and average velocity from Eq. (2), with a scale multiplierfor the length of the putter for speed at the putter head at impact witha ball, see Eq. (1). For long putts the acceleration method becomesincreasingly inaccurate, hence the average velocity method providesbetter results with a multiplier derived from empirical measurements.

From the speed of the putter head the distance the ball travels can becalculated assuming ideal conditions. Most important, however, is thatwe are able to quantify phone roll angle differences at impact (similarto hook or slice for the full swing) without impacting a physical ball.We can also analyze the gyroscope acceleration data for errors such asdeceleration through the putt, or a left pull or right push (these lasttwo errors are identified from the combination of the second integral ofacceleration, and the roll data). Data on swing motion accuracy is alsopresented to the user and stored, local to the app and on the server inthe user's account, for longitudinal comparisons of putting consistencyimprovement.

Multi-Sensor Impact Detection

A technique for detecting the “impact point” with a virtual object usinga single type of rotational data (pitch) of the mobile device 10 wasdescribed above. The signature of the sports motion (pitch data as afunction of time) was analyzed for characteristics, specific to the typeof sports motion (e.g., a full golf swing). The a priori structure ofthe sports motion signature was necessary to isolate the location intime and space of the virtual impact point. In another embodiment, weextend our inventive concept to enable impact point detection for manydifferent sports motion signatures, and for a wide range of motions.

FIG. 6 illustrates changes in pitch and yaw of the mobile device duringan example full golf swing. In this case, the mobile device used was anApple iPhone 4Gs. As noted above, calibration of the mobile device 10 isaccomplished by holding the mobile device 10 still in the addressposition (position 1). The motion signature for the pitch then increasesin the backswing, (position 2) and has a local minimum at the top of thegolf backswing (position 3). However, the minimum (position 3) is anartifact of the pitch motion sensor rotating more than 180 degrees. Asnoted previously, in actuality, the pitch continues to increase to amaximum, greater than 180 degrees, at the top of the backswing. However,limitations of the sensor constrain the motion signature to 0 to 180degrees. The pitch data continues to decrease in the downswing (position4), back to the impact point (position 5), as shown.

From detailed experiments with high speed cameras we found that thevirtual impact point (position 5) is a local minimum of pitch, where themobile device has returned near to the initial address position(position 1). From the impact point (position 5), the golf swingcontinues through follow through (positions 6 and 7).

Determining the impact point is of vital importance because the roll ofthe phone at this point defines the hook or slice of the club, bat orracquet, and/or the release point in throwing or casting sports. Theinventors have previously used crawler software to search the pitchmotion signature for the second minimum. However this method is notuniversally applicable, since different swings have different motionsignatures.

FIGS. 7( a) to (c) illustrates changes in pitch of the mobile device forthree different types of golf swings. FIG. 7( a) shows a full golf swingof a professional golfer, 7(b) a golf chip, and 7(c) a golf putt. Whilethe impact point is the same in all three cases, the motion signaturesare quite different. Furthermore, even in a full golf swing, the basicmotion signature can be different. Specifically, it was found that forolder people playing golf, there is a tendency to abbreviate the backswing, so that the swing signature looks more like a chip.

Hence, the crawler method which searches for a specific feature of themotion signature of a single motion sensor output produces erroneousresults. Specifically, in the case of golf, the motion signature for aprofessional golfer's full swing has an impact point at the secondminimum of pitch data. However, FIGS. 7( b) and (c) do not have a secondminimum; hence searching for the second minimum in these types of shotswill create an error. Accordingly, the method of using motion signaturedata for single type of rotational measurement to obtain the impactpoint has limitations. In the present embodiment, we use at least twodifferent types of rotational measurements (pitch and yaw in golf forexample) to calculate the impact point and/or release point to overcomethis.

Referring again to FIG. 6, the yaw of the mobile device through the golfswing is shown. In the case of golf and baseball swings, the yaw ischanging rapidly through the impact point (5). FIG. 6 also shows thecorresponding derivative, or slope, of the yaw. These data quantify therate of change of the yaw sensor data. Note that the maximum rate ofchange is close to the impact point (5) for the golf swing. Hence, usingboth pitch and yaw motion sensor data, one can isolate the impact zoneby looking for the minimum of pitch motion data that has a maximum yawderivative (change in yaw). This method works for all types of golfswings, and enables the accurate impact point detection for chips andputts, such as those shown in FIGS. 7( b) and (c).

This technique is generalizable to other types of sports motions. FIG. 8is an example of a tennis forehand, or a seated horizontal golf swing.In this example the swing path is in the horizontal plane but withforearm rotation and wrist hinging around impact. Hence, the motionsignature is different from a standing golf swing and the impact pointin FIG. 8 for pitch is now a zero crossing of pitch data. The challengeis to detect the correct zero crossing. In this example, the yaw is alocal maximum near the impact point. Hence again using two types ofrotational measurements (pitch and yaw), see FIG. 8 “pitch” and “yaw”,we can more accurately, and less erroneously, detect the impact pointfrom the single sensor, in this case pitch. In the case of a tennisswing, see FIG. 8 “roll”, the roll data at the impact point can be usedto calculate the hook or slice spin imparted to the tennis ball.

Baseball and Bowling Examples

To illustrate preferred embodiments where the sports motion may (1)intersect with a moving virtual object and where (2) the release pointand impact point is different from the calibration point, we provideexamples for baseball and bowling.

Baseball swing motion sensor data is illustrated in FIG. 9. For abaseball swing the calibration point is a set-up position with themobile device 10 held in both hands out in front of the body, with thethumbs pointing so as to naturally line up the mobile device (virtualbat) with a ball on a virtual tee; the hands are perpendicular to theground. The data shown in FIG. 9 is from a professional athlete andillustrates the essential features of an optimal baseball swing motion.For the baseball sports motion, yaw is the key variable, since as the“bat” is swung through the impact point with a virtual ball, the idealhand position is with the palms parallel to the ground, which causes arapid change in yaw of the mobile device through impact. The yaw at thecalibration point was zero; hence the impact point is when the yawcrosses zero (see, FIG. 9, “yaw”), even though the mobile device isrotated ninety degrees relative to the calibration point. In an idealbaseball swing the roll of the bat occurs just after the impact point(see, FIG. 9, “roll”). In the event there is a roll maximum at theimpact point, then the wrists have a tendency to lift the bat over thetop of the ball, causing a ground or missed ball: this is the “swingbubble.”

The pitch and yaw of the mobile device 10 taken together provideinsights into the angle of the bat through the impact point. Forexample, the pitch data in FIG. 9 shows that the hands sloped downwardat the impact point, since the pitch is negative at the impact point anddoes not return to zero until after the impact point, and hence the batwould have contacted the virtual ball if it were thrown below thecalibration point, that is, in the lower half of the strike zone.

Jeffery et al., 61/580,534 and Jeffery et al 61/641,825 have disclosed amethod using multiple displays wherein virtual sports instruction and/orgames can be played using a web-enabled display device, such as aweb-enabled TV, that is separate and distinct from the mobile device,and that coordinates presentation of the virtual sports instructionand/or games using the mobile device and the web-enabled display devicevia a cloud-based software engine. Accordingly, animations, lesson andother video may be presented on the display device physically separatedfrom the mobile device and responsive to the mobile device motion sensoroutputs, for example.

As an example of this embodiment for baseball, the player (1) stands infront of their HTML5 Web-enabled TV and calibrates the mobile device asabove. (2) They then see a displayed video or an animation (which caninclude a sprite or a cinema-graph, or other visual enhancement) of apitcher throwing the ball, the screen of the web-enabled TV positionedso that the pitcher appears directly in front of the hitter. Thecloud-based software engine (3) synchronizes the time of the pitch andcompares it to that of the swing (4) of the player, and the playersmotion sensor data is analyzed on the mobile device and is sent to thecloud-based software engine.

The time t_(ball flight), it takes for the virtual pitch to reach theplayer can be calculated from t_(ball flight)=d/v where d is thedistance from the pitcher to home plate (60.5 feet for major leaguebaseball or 45 feet for little league, as examples) and v is thevelocity of the pitch. Assuming a 95 mph pitch in major league baseball,the time of flight of the baseball from the pitcher to home plate is0.43 seconds. That is, t_(ball flight)=0.43 seconds. The cloud-basedengine compares the time stamp of the thrown pitch, t_(pitch), plust_(ball flight) to the time stamp of the impact point, t_(impact point).If they coincide within a predetermined time intervalΔt=|t _(impact point)−(t _(pitch) +t _(ball flight))|  (5)less than or equal to δ seconds, 0.15 seconds for example, then thevirtual bat can be assumed to have hit the virtual ball, and (6) ananimation of the ball flight can then be rendered on the Web-enableddisplay via the cloud-based software engine. However, if Δt>δ seconds,the virtual bat is assumed to have missed the virtual ball and the swingis deemed a strike.

Preferably, the sports motion analysis and synchronization uses asynchronized mobile device 10, a cloud-based (or otherwise networked)software engine, and a Web-enabled display each with a fidelity of 0.1seconds or less. Current web browsers have unreliable local clock timestamps and Javascript calls to the internal clock typically do not pollat exactly equal intervals. In a preferred embodiment, the Network TimeProtocol (NTP) can be used to synchronize the computer systems over apacket-switched, variable-latency data network. We use the Java ScriptNTP client to acquire the time offsets of the clients (mobile device 10and web page) and the server (cloud based software engine). This setsthe initial coordinated time based upon an accurate external clock. Wethen schedule a Java Script callback using setInterval( ) at the highestreliable granularity possible, which is web browser dependent. We do notassume that the callback is being called at reliable intervals, however,but instead use the call new Date( ).getTime( ) from within the callbackand apply the offset to get the actual coordinated time, and theninterpolate to find the actual time of the pitch, t_(pitch), and thevirtual impact point, t_(impact point). These data are then used tocalculate Eq. (5).

Hence our method is generalizable and extensible to the use case wherethe sports motion is impacting a moving virtual object, such as abaseball or tennis ball, and can be similarly applied to tennis,badminton, table tennis, racquet ball, hockey, basketball, Americanfootball, and all other similar sports where the virtual sports object(e.g., ball, puck, shuttlecock) is in motion and then struck, thrown, orcaught by the sports motion and the players virtual sports equipment.

As a last example, we consider the use case where the release point isdifferent from both the calibration point and the impact point. FIG. 10illustrates the mobile device motion sensor data for a bowling sportsmotion. In this example, the calibration point is the hand at rest,relaxed and fully extended at the player's side, with the palm of thehand facing forward. The bowling motion is to first bring the virtualbowling ball up to the chin, cradled in both hands, and then to swingdown and forwards while taking a few steps. The pitch data illustrateshow the pitch of the mobile device 10 increases as the mobile device 10is brought up to the chin, where there is a local minimum as the playerstarts to step forward. Then, the pitch decreases as the player swingsdown in the backswing motion, where there is a zero of pitchcorresponding to the initial calibration zero, and then the motiontransitions to the final downswings to a second zero, which is therelease point of the virtual bowling ball.

Similar to the golf swing described previously, the velocity of thevirtual bowling ball can be calculated from Eq. (2) and the timedifference between 30 or 60 degree pitch points, similar to FIG. 5, orvia integrating Eq.'s (3). The rate of change of the roll data, thederivative of roll, through the release point is proportional to thespin rate imparted to the virtual bowling ball. Hence we can calculatethe velocity and spin of the virtual bowling ball at the release point.

Note in this example the release point is different in space from thecalibration point, and the impact point is further removed from therelease point. In this example, the impact point occurs in virtualspace. Using a cloud-based system described previously for baseball, thebowling ball can be displayed on a virtual bowling lane on an HTML5web-enabled display, such as a web TV, and the impact with the pinssimulated in time and space given the velocity and spin of the virtualbowling ball, and the length of the virtual bowling lane. Hence, theplayer executes the virtual bowling motion, and sees the virtual bowlingball go down the lane and hit the pins on the Web-enabled display, witha path and speed determined by the velocity and spin calculated from theswing signature of the mobile device and synchronized in time to appearlike a continuous motion.

Attachment to an Ancillary Device

Thus far, the description of the invention has been limited to use ofthe mobile device 10 to simulate a sports motion by the user holding themobile device 10 in his or her hand and moving the mobile device 10 in acertain manner (e.g., swinging the mobile device 10 as if it were a golfclub). However, advanced players may find it desirable to feel the gripof the sports equipment in sports such as golf, baseball, tennis or flyfishing, for example. In the case of golf, for a right handed player,advanced players may have a grip on the club so that the left hand isrotated approximately 20 degrees from center towards the body. Such agrip on the golf club handle enables the club head to be more closedthrough impact which in turn makes it easier to draw the golf ball, thatis, create a ball flight that bends to the left.

The methods of the present invention relating to analysis of sportsmotions are generalizable to also include attachment of the mobiledevice to sports equipment, or to weighted grips simulating the sportsequipment.

As an example, FIG. 11 shows a mobile device holder 20 to securely mountthe mobile device 10 to an ancillary device 30, which in the illustratedembodiment is a weighted golf grip but could instead (for golf) be aphysical golf club. In an embodiment, the ancillary device 30 iscomprised of a 24″ long steel or graphite golf club shaft with a golfgrip at one end and a 6 ounce weight 18 at the other. Preferably, theentire ancillary device 30 weighs approximately 11 ounces (similar to agolf club driver), and the center of mass is approximately 6-8″ inchesfrom the weight 18, so as to simulate an actual golf club, whichtypically has the center of mass approximately ¼-⅓ the length of theshaft closer to the club head. FIG. 11 is presented for illustrativepurposes, and is not meant to be limiting. Other sports, such asbaseball, tennis, and fly fishing, would have different ancillarydevices but the grip, weighting and center of mass more accuratelysimulate the actual sports equipment, and/or the mobile device could beattached to the actual sports equipment via the holder 20.

Referring to FIG. 12( a), the mobile device holder 20 comprises atwo-piece assembly including a C-shaped coupler 24 and a frame 22. Asshown, the frame 22 is rectangular and includes a pair of grooved sides25 and an open end 27. The C-shaped coupler 24 is structured so as tosnugly fit around the ancillary device 30 (as shown). As shown, theframe 22 includes hole 23. The hole 23 is sized to accommodate collarwings 26 of the C-shaped coupler 24, which can be fitted through thehole 23 such that the frame 22 is positioned perpendicularly relative tothe longitudinal axis of the ancillary device 25, as shown in FIG. 12(b). Next, the frame 22 is turned 90 degrees such that the open end 27points away from the club head, as shown in FIG. 12( c). Once turned 90degrees, the collar wings 26 settle into pockets 28 molded on the insideof the frame 22. Once the collar wings 26 are seated, the mobile device10 can be slid into the frame, the grooved sides 25 providing a securefriction fit. In an embodiment the materials used for the frame 22include a hard polycarbonate, most preferably, co-molded silicontogether with the polycarbonate for an enhanced friction fit with themobile device 10. However, it is to be understood that various othermaterials may suffice, such as stainless steel, aluminum, or anothermetal; polyethylene, acrylonitrile-butyl-styrene (ABS), polyvinylchloride, and nylon, or another plastic. Further it is to be understoodthat the particular manner in which the mobile device 10 is mounted tothe ancillary device 30 (i.e., using the mobile device holder 20) ispresented for illustrative purposes, and is not meant to be limiting.

As mentioned, an important feature of the present invention is thatimpact with a physical sports object, such as a golf or tennis ball, isnot required. However, the player may, in various embodiments, attachthe mobile device 10 to an ancillary device to hit physical balls. As anexample detailed experiments were conducted with mobile devices 10attached to actual sports equipment where professional athletes hitphysical sports objects in order to validate the methods described inthis invention.

While this invention has been described in conjunction with the variousexemplary embodiments outlined above, it is evident that manyalternatives, modifications and variations will be apparent to thoseskilled in the art. Accordingly, the exemplary embodiments of theinvention, as set forth above, are intended to be illustrative, notlimiting. Various changes may be made without departing from the spiritand scope of the invention.

What is claimed is:
 1. A method of analyzing sports motions, comprising:(a) determining a starting point of a sports motion to be simulatedusing a mobile device having integrated motion sensors, wherein thestarting point is indicated solely by a user holding the mobile devicestill for a predetermined length of time; (b) moving the mobile devicefrom the starting point along a path to complete the simulation; (c)obtaining motion data from the motion sensors relating to the startingpoint and the movement along the path; (d) determining an occurrence ofa simulated sports event using the obtained motion data; and (e)outputting information related to the simulated sports motion based atleast in part on the simulated sports event, wherein the steps (c) to(e) are performed using a processor integral to the mobile device; (f)wherein the mobile device is not attached to any piece of sportsequipment and the starting point is not indicated by user-entered input.2. The method of claim 1, wherein the mobile device is a phone.
 3. Themethod of claim 1, wherein outputting the information related to thesimulated sports event includes outputting the information on a screenof the mobile device.
 4. The method of claim 1, wherein outputting theinformation related to the simulated sports event includes outputtingthe information to a gaming system.
 5. The method of claim 1, whereinthe motion sensors include an accelerometer and a multi-axis gyroscope.6. The method of claim 1, wherein the sports event is an impact pointwith a virtual object.
 7. The method of claim 6, further comprising thestep of determining the velocity of a virtual object around the impactpoint.
 8. The method of claim 7, wherein determining the velocity isbased at least in part on determined velocity of the mobile devicearound the impact point, determined arm length, determined club length,and determined arc length for a determined swing type.
 9. The method ofclaim 7, wherein determining the velocity does not include use of datafrom an accelerometer.
 10. The method of claim 7, wherein determiningthe velocity includes calculating the velocity using first motion dataand applying a multiplier based on simultaneous second motion data. 11.The method of claim 6, further comprising determining the instantaneousvelocity of a virtual object at the impact point.
 12. The method ofclaim 6, wherein the impact point is determined by finding a moment whena first rotational orientation of the mobile device is at or near aminimum and the derivative of a second rotational orientation of themobile device is at or near a maximum.
 13. The method of claim 6,wherein the motion sensors include a multiaxis gyroscope, and the impactpoint is determined using two or more of pitch, roll, and yaw.
 14. Themethod of claim 1, further including analyzing the simulated sportsmotion.
 15. The method of claim 14, wherein analyzing the simulatedsports motion includes analyzing one of the pitch, roll and yaw of themobile device during the simulated sports motion.
 16. The method ofclaim 14, wherein the simulated sports motion is determined based on afirst motion of the mobile device during the simulated sports motion anda second motion of the mobile device at an impact point.
 17. The methodof claim 16, wherein analyzing the second motion of the mobile device atthe impact point includes subtracting the second motion at the impactpoint from the second motion at the starting point.
 18. The method ofclaim 1, wherein the sports event is a release point.
 19. The method ofclaim 1, wherein the release point is one of a release point of abowling ball, a lacrosse handle, a basketball, a baseball, a hockeystick, a bean bag, an American football and a fishing rod.
 20. Themethod of claim 1, wherein outputting information related to thesimulated sports motions includes outputting the information using atleast one of the mobile device and a separate web-enabled display. 21.The method of claim 1, wherein outputting information related to thesimulated sports motions includes outputting one or more of a renderedgolf ball flight, a baseball flight, a basketball flight, a tennis ballmovement, a ping pong ball movement, a bean bag toss, an Americanfootball flight, a bowling ball path, a fishing hook flight and othermotion responsive to the simulated sports motion.
 22. A system foranalyzing sports motions, comprising: a mobile device including memory,a processor, an integrated multi-axis gyroscope, and a cloud-basedengine which connects the mobile device and a web-enabled display withthe Internet and manages game playing, and an analysis engine thatincludes an application program loaded in the memory of the mobiledevice, the mobile device configured to: (a) determine a starting pointof a simulated sports motion, wherein the starting point is indicatedsolely by a user holding the mobile device still for a predeterminedlength of time; (b) obtain gyroscope measurements from the gyroscoperelating to orientation of the mobile device at the starting point andduring movement along a path simulating the sports motion; (c) determinean impact point with a virtual object and velocity of the mobile devicearound the impact point using the obtained gyroscope measurements; and(d) output information related to the simulated sports motion based atleast in part on the impact point and the velocity.
 23. The system ofclaim 22, wherein the mobile device is attached to an ancillary device.24. The system of claim 23, wherein the ancillary device is a piece ofsports equipment and a device simulating the grip and feel of the pieceof sports equipment.
 25. A system for analyzing sports motions,comprising: a mobile device including memory, a processor, integratedmotion sensors, a cloud-based engine which connects the mobile deviceand a web-enabled display with the Internet and manages game playing,and an analysis engine that includes an application program loaded inthe memory of the mobile device, the mobile device configured to: (a)determine a starting point of a simulated sports motion, the startingpoint is indicated solely by a user holding the mobile device still fora predetermined length of time; (b) obtain motion data from the motionsensors relating to orientation of the mobile device at the startingpoint and during movement along a path simulating the sports motion; (c)determine an impact point with a virtual object using the obtainedmotion data related to the orientation of the mobile device along atleast two different axes; and (d) output information related to thesimulated sports motion based at least in part on the impact point viathe mobile device.
 26. The system of claim 25, wherein the mobile deviceis attached to an ancillary device.
 27. The system of claim 25, whereinthe ancillary device is one of a piece of sports equipment and a devicesimulating the grip and feel of the piece of sports equipment.
 28. Themethod of claim 1, wherein the starting point is indicated by the mobiledevice emitting a sound or vibration.
 29. The method of claim 6, whereinthe impact point is found at a minimum of one of pitch, roll or yaw ofthe mobile device along the path.
 30. The method of claim 6, wherein thevirtual object is a virtual golf ball.
 31. The method of claim 7,wherein the virtual object is a virtual golf ball.
 32. The method ofclaim 31, wherein determining the velocity is based at least in part ondetermined velocity of the mobile device around the impact point,determined arm length, determined club length, and determined arc lengthfor a determined swing type.
 33. The method of claim 31, furtherincluding determining ball flight distance based at least in part on thedetermined velocity of the virtual golf club head.
 34. The method ofclaim 31, wherein determining the velocity includes applying amultiplier based on estimated wrist hinge and forearm rotation asmeasured by yaw and roll of the mobile device at the impact point. 35.The method of claim 1, wherein moving the mobile device from thestarting point along the path includes swinging the mobile device tosimulate a golf swing.
 36. The method of claim 35, wherein the golfswing is one of a full golf swing, a chip, and a putt.
 37. The method ofclaim 30, wherein the impact point is determined by finding a momentwhen pitch of the mobile device is at or near a second minimum.
 38. Themethod of claim 14, wherein analyzing the simulated sports motionincludes analyzing the pitch of the mobile device during the simulatedsports motion.
 39. The method of claim 17, wherein the second motiondata is roll of the mobile device.
 40. The method of claim 6, whereinthe impact point is determined by finding a moment when a firstrotational orientation of the mobile device is at or near a minimum andthe derivative of a second rotational orientation of the mobile deviceis at or near a maximum.
 41. The method of claim 40, wherein the firstrotational orientation of the mobile device is pitch and the secondrotational orientation of the mobile device is yaw.
 42. The method ofclaim 2, wherein the screen of the phone is substantially perpendicularto the sports motion path at the calibration point of the sports motion.43. The method of claim 42, wherein the sport is one of golf, baseball,basketball, tennis, bowling, ping pong, and bean bag tossing.
 44. Themethod of claim 1, wherein moving the mobile device from the startingpoint along the path includes swinging the mobile device to simulate oneof a golf swing, a baseball swing, a bowling motion, a tennis swing, abadminton swing, and casting of a fishing rod.